Data integration and Data pre

1. Aim

The purpose of assignment 2 is to prepare a dataset with which you can conduct data analyses for the term project. This is related to the term project, but please note that this assignment should be individual work. Also, you need to explore some variables that can be used as a dependent variable and independent variables. More details will be explained below.

Dataset

For the project phase 1 report, you already selected a medical condition of your group’s interest, and there should be more than 500 data instances. For the assignment 2, please think about your research question(s). With that being said, you need to explore variables in your dataset. You can change those variables later, but this assignment will help you better understand the dataset. Please visit the official website (Links to an external site.) and Github repository (Links to an external site.) and read the Download attached appendix document(p.1 – 13). MEPS data consists of various variables such as medical condition, socioeconomic factors (e.g., gender, region, race, and family income), and medical expenditure. MEPS data also consist of various files such as person-level (e.g., health status, demographics, and total $$ of care), event-level (e.g., healthcare service use), and condition-level (e.g., medical condition). For the full review of those variables, please look at codebooks (person-level (Links to an external site.)) and condition-level (Links to an external site.)). I also coded those variables (personal-level only) regarding usefulness for analysis (included vs. excluded, Heejun_Inclusion field) and variable type (independent vs. dependent, Heejun_Variable_Type field). You can find my version of the codebook from this link (Links to an external site.). Some dependent variables you can utilize includes but are not limited to:

  • Total health expenditures
  • Total inpatient expenditures
  • Total emergency care expenditures
  • Severity of Illness (attacks/year)
  • Number of School Days Missed (Children)
  • Number of Work Days Missed (Adult)

For now, just focus on personal-level (h129.csv) and condition-level (h128.csv) files for exploring variables. At least, find one dependent variable and five core independent variables that can be used for the group project by carefully looking at documents and websites above. The selected independent variables should not be randomly selected. They should be clinically related to your dependent variable.

The second step is to preprocess your dataset. Please utilize the “week6_example_code.ipynb” notebook for this process. You need little modification to complete data processing and integration. Select patients (h128.csv) with the your medical condition, drop attributes which are not necessary, and join the condition-level data (h128.csv) and personal-level data (h129.csv).

What to Do for the Assignment 2

  1. Explore the dataset and find one dependent variable and five core independent variables at least
  2. Explain what these variables are and how they are related to the medical condition of your interest
  3. Preprocess your data. The preprocessing will include but are not limited to:
    • h128.csv
      • Select records that are related to your condition of interest by using ICD-9 code and drop duplicated records based on the ‘DUPERSID’ (rowwise selection)
    • h129.csv
      • Select attributes that are discriminative and/or meaningful (columnwise selection)
    • Join two files (h128.csv and h129.csv) after selecting patient instances with the medical condition of your interest

How to write

For the assignment 2, I will not put constraints on the format. However, you need to present, at least, one dependent variable and five core independent variables. Also, please explain what these variables are and how they are related to the medical condition of your interest. Up to this part, you can write the report in a MS word file. You also need to submit your Jupyter notebook you used for data preprocessing and the final data file (csv).

What to include

Your submission of the report should include:

  • Your name
  • EUID
  • Title of the project
  • What I asked you in the “How to Write”

Please attach a .docx file for the report and .ipynb and .csv files for the data processing. Again, this is an individual assignment.

Grading Criteria

Your reports will be evaluated on the quality of entries. By quality, I do not mean mechanics (e.g., spelling, grammar), but rather content. Your entries should demonstrate reflective and critical thinking, integration of materials and experiences, development of your own research questions, and general intellectual growth. Also, I will look at the data preprocessing steps you implemented.

2.Aim

The purpose of assignment 3 is to preprocess the dataset with which you can conduct data analyses for the term project. This is related to the term project. Please note that this assignment should be group work. Basically, you are expected to 1. find any indiscriminative (not meaningful) variable(s) and remove them, 2. process missing values, 3. remove outliers, 4. create a new attribute by using existing attributes as necessary, and 5. normalize data.. More details will be explained below.

Dataset

You need to use a dataset you created for assignment 2. As individual group members did their own work, it is important for you to discuss what variables you are going to use for assignment 3. You may want to include some new variables to practice with removing redundancy. For instance, If you only had “RACEX” in the dataset, it would be good to include “RACETHNX” to examine whether they are redundant or not. Modifying your previous notebook and creating a new dataset that is different from your previous submission is totally fine. Data analysis is an iterative process.

What to Do for the Assignment 3

  1. Do practice with the week 11 example code
  2. Find indiscriminative variables
    • You may find or not indiscriminative variables. Please provide your rationale why they are discriminative or indiscriminative.
  3. Process missing data
    • Some of your variables would have negative values. For some reason, those data are missed. Please process (e.g., ignore or replace with new values) these missing data and provide your rationale for selecting the method your group chose. Also, please explain some pros and cons of your approach.
  4. Find outliers and remove them
    • Please explain what thresholds you used
  5. Create new attributes as necessary (e.g., merging “BMINDX53″ and”CHBMIX42”). You can skip this step if you don’t need to create new attributes.
  6. Check redundancy of attributes by using correlations tests and remove redundant attributes.
    • Please provide test results (e.g., Person’s correlation coefficient for numeric data and Pearson’s χ2 correlation coefficient, degree of freedom, and critical values for categorical data) to support your decision.
    • You may not find redundant attributes which are totally fine. In this case, you have to provide evidence (i.e., test results).
  7. Normalize data

How to write

For the assignment 3, I will not put constraints on the format. However, you need to explain what you have done for data processing in high-level. Also, please provide rationale for your decision and supporting evidence (e.g., correlation coefficients or descriptive statistics) (refer to What to Do for the Assignment 3). Up to this part, you can write the report in a MS word file. You also need to submit your Jupyter notebook you used for data preprocessing and the final data file (csv).

What to include

Your submission of the report should include:

  • Names of students in your project group
  • EUIDs of group members
  • Title of the project
  • What I asked you in the “How to Write”
  • The order of contributions (i.e., work distribution) like authorship (e.g., first author, second author, third author, and so on.)
    • If you believe that some of you or all of you contributed equally, then you need to state it
    • Here, contributions include all phases you worked to submit this project report

Please attach a .docx file for the report and .ipynb and .csv files for the data preprocessing.

Grading Criteria

Your reports will be evaluated on the quality of entries. By quality, I do not mean mechanics (e.g., spelling, grammar), but rather content. Your entries should demonstrate reflective and critical thinking, integration of materials and experiences, development of your own research questions, and general intellectual growth. Also, I will look at the data preprocessing steps you implemented.


Order the answer to view it

Assignment Solutions


Assignment Solutions

ORDER THIS OR A SIMILAR PAPER AND GET 20% DICOUNT. USE CODE: GET2O

Posted in Uncategorized