10 Essential 数据分析 面试问题 *

There are plenty of alternatives to h和le missing data, although none of them is perfect or fits all cases. 其中一些是:

• 删除不完整行: Simplest of all, it can be used if the amount of missing data is small 和 seemingly r和om.
• 把变量: This can be used if the proportion of missing data in a feature is too big 和 the feature is of little significance to the analysis. 一般来说，应该避免使用它，因为它通常会丢掉太多的信息.
• Considering “not available” (NA) to be a value: Sometimes missing information is information in itself. 取决于问题域, missing values are sometimes non-r和om: Instead, they’re a byproduct of some underlying pattern.
• 价值归责: This is the process of estimating the value of a missing field given other information from the sample. There are various viable kinds of imputation. Some examples are mean/mode/median imputation, KNN, regression models, 和 multiple imputations.

在任何面向数据的进程中，“垃圾输入，垃圾输出”问题总是可能出现的. 为了减轻它, 我们利用数据验证, a process composed of a set of rules to ensure that the data reaches a minimum quality st和ard. A couple of examples of validation checks are:

• 数据类型验证: Checks whether the data is of the expected type (eg. integer, string) 和 conforms to the expected format.
• Range 和 constraint validation: Checks if the observed values fall within a valid range. 例如, temperature values must be above absolute zero (or likely a higher minimum depending on the operating range of the equipment being used to record them.)

Linear regression is a statistical model that, 给定一组输入特征, 尝试拟合最好的直线(或超平面), (一般情况下)在自变量和因变量之间. 自 its output is continuous 和 its cost function measures the distance from the observed to the predicted values, it is an appropriate choice to solve regression problems (e.g. 预测销售数字).

逻辑回归, 另一方面, 输出一个概率, 根据定义，哪个是0到1之间的有界值, due to the sigmoid activation function. 因此，最适合解决分类问题(如.g. 来预测一个给定的交易是否欺诈).

Model extrapolation is defined as estimating beyond a previously observed data range to establish the relationships between variables.

外推法的主要问题在于，它充其量只是一种有根据的猜测. 自 it has no data to support it, 通常不可能声称观察到的关系仍然成立. A relationship that looks linear in a given range might actually be non-linear when outside of range.

Data leakage is the process of training a statistical model with information that would be actually unavailable when using the model to make predictions.

Data leakage makes the results during model training 和 validation much better than what is observed when the model is deployed, generating too optimistic estimates, possibly leading to an entirely invalid predictive model.

There is no single recipe to eliminate data leakage, but some practices are helpful to avoid them:

• Don’t use future data to make predictions of the past. 虽然明显, it’s a very common mistake when validating models, especially when using cross-validation. When training on time-series data, always make sure to use an appropriate validation strategy.
• Prepare the data within cross-validation folds. Another common mistake is to make data preparations, like normalization or outlier removal on the whole dataset, prior to splitting the dataset to validate the model, 哪个是信息泄露.
• 调查id. It’s easy to dismiss IDs as r和omly generated values, 但有时它们会编码目标变量的信息. 如果它们是漏的，最好从任何类型的模型中移除它们.

答案不是封闭的，而是取决于以前的经验和领域专业知识. The goal is not to get every single item right, but to showcase critical thinking 和 domain knowledge.

对于这种情况，可以探索的一些可能路径是:

• Determine which are the most popular items sold
• Explore how much is spent per transaction
• Find which clients spend the most
• Find the most recurrent clients
• Uncover seasonalities 和 trends

All of the above can be analyzed over the whole dataset or by region, by store or by time frame. The analyses could be further enriched with store, client 和 item information, if available.

The information uncovered could be used to:

• Better scale inventory sizing 和 the number of on-site employees using time-series forecasting.
• Perform direct marketing to the most profitable clients, 哪些可以通过聚类技术识别.
• 通过将可能一起购买的商品分组，提高商品在商店中的定位, which could be identified through market basket analysis. Recommender systems could also be applied.

1. Finding a relevant business problem to solve: 常被忽视, this is the most important step of the process, 因为产生业务价值是任何数据分析师的最终目标. Having a clear objective 和 restricting the data space to be explored is paramount to avoiding wasting resources. 自 it requires deep knowledge of the problem domain, 此步骤可能由数据分析人员以外的领域专家执行.
2. 数据提取: The next step is to collect data for analysis. It could be as simple as loading a CSV file, 但通常情况下，它涉及到从多种来源和格式收集数据.
3. 数据清理: 数据收集完成后，需要准备数据集进行处理. Likely the most time-consuming step, data cleansing can include h和ling missing fields, 破坏数据, 离群值, 重复的项.
4. 数据探索: 在考虑数据分析时，这通常会出现在脑海中. Data exploration involves generating statistics, 特性, 以及数据的可视化，以更好地理解其潜在模式. 然后，这会产生可能产生业务价值的见解.
5. Data modeling 和 model validation (optional): 培训一名统计或 机器学习 模型并不总是必需的, as a data analyst usually generates value through insights found in the data exploration step, but it may uncover additional information. 易于解释的模型, like linear or tree-based models, 和 clustering techniques often expose patterns that would be otherwise difficult to detect with data visualization alone.
6. 讲故事: This last step encompasses every bit of information uncovered previously to finally present a solution to—or at least a path to continue exploring—the business problem proposed in the first step. It’s all about being able to clearly communicate findings to stakeholders 和 convincing them to take a course of action that will lead to creating business value.

These are the most common steps of data analysis. Although they have been presented as a list, more often than not they are not executed sequentially 和 some steps may require several iterations as new data sources are added 和 information is uncovered.

Correlation is a statistic that measures the strength 和 direction of the associations between two or more variables.

另一方面，因果关系是一种描述因果关系的关系.

“Correlation does not imply causation” is a famous quote that warns us about the dangers of the very common practice of looking at a strong correlation 和 assuming causality. 在下列情况下，强相关性可能没有因果关系:

• 隐藏变量: 影响两个感兴趣变量的未观察变量, causing them to exhibit a strong correlation, even when there is no direct relationship between them.
• 混杂变量: A confounding variable is one that cannot be isolated from one or more of the variables of interest. Therefore we cannot explain if the result observed is caused by the variation of the variable of interest or of the confounding variable.
• 伪相关: 有时由于巧合, 即使没有合理的逻辑关系，变量也可以相互关联.

Causation is tricky to be inferred. 最常见的解决方法是进行随机实验, 在那里作为候选原因的变量被隔离和测试. 不幸的是, 在许多领域进行这样的实验是不切实际的或不可行的, 因此，运用逻辑和领域知识对于得出合理的结论至关重要.

精确率和召回率是衡量分类性能的指标, 每个都有自己的标准, 由下式给出:

\[\text{Precision} = \frac{TP}{TP+FP}\] \[\text{Recall} = \frac{TP}{TP+FN}\]

地点:

TP =真阳性
FP =假阳性
FN =假阴性

换句话说, 精度 is the ratio of correctly classified positive cases over all cases predicted as positive, 召回率是正确分类的阳性病例与所有阳性病例的比率.

当假阳性的代价很高时，精度是一个合适的度量.g. email spam classification), while 回忆 is appropriate when the cost of a false negative is high (e.g. 欺诈检测).

两者也经常以f1分数的形式一起使用，其定义为:

\[\text{F1} = 2*\frac{\text{Precision} * \text{Recall}}{\text{Precision}+\text{Recall}}\]

The F1-score balances both 精度 和 回忆, 因此，对于高度不平衡的数据集，这是一个很好的分类性能度量.

通常, 数据通过使用图像中的位置(高度)的图表直观地表示, 宽度, 和深度). Going beyond three dimensions, we need to make use of other visual cues to add more information. 最常见的有:

• Color一种视觉上吸引人且直观的方式来描述连续和分类数据.
• 大小: Marker 大小 is also used to represent continuous data. Could be applied for categorical data as well, 但由于尺寸差异比颜色更难察觉, 对于这种类型的数据，它不是最合适的选择.
• 形状最后，我们有形状，这是一个有效的方式来表示不同的类.

结合以上所有，我们可以可视化到六个维度, though one could argue that cramming so much information in a single chart does not make for a very effective visualization.

Another possibility is to make an 动画 图表，这是非常有用的描述变化随着时间的推移: