Skip to content

sharkness/FEC-Data-Wranglin

 
 

Repository files navigation

This project is rapidly evolving and the README will be updated soon.

FEC-Data-Wranglin

get_that_data.py pulls data from FEC API clean_that_data.py finds duplicate values from the same column using TF-IDF i.e. "Not Employeed" is replaced by "Not Employed" or "Apple INC" is replaced with "Apple"

If you want to make more than a handful of requests you need an api_key, visit fec.gov to get a key. Set your api_key as an environment variable. i.e. FEC_API_KEY=DEMO_KEY

When you run python3 get_that_data.py in the terminal:

Resulting data will be saved in a csv in FEC-Data-Wranglin/data/raw_data and will be structured as seen below.

Each row contains the information of one donation, the first five columns reference the contributor's information and party is the party of the candidate they donated to.

contributor_occupation contributor_employer contributor_city contributor_state contributor_zip party
0 RETIRED RETIRED BENTON AR 72019 OTH
1 SYSTEMS MANAGER AVANIR PHARMACEUTICALS ALISO VIEJO CA 92656 OTH
2 RETIRED RETIRED OSHKOSH WI 549048984 OTH
3 INSURANCE REP BRS FINANCIAL GROUP FRESNO CA 93701 IND
4 EDITOR GLOBAL FINANCE MEDIA GREENLAWN NY 11740 OTH
5 FALSE FORMATIV HEALTH HOOSICK FALLS NY 12090 OTH

After gathering your data you can try to clean it up a little.

The process I have used for cleaning the data can be found here

The more data you have the better the cleaning will work!

Run python3 clean_that_data.py after python3 get_that_data.py.

You can change the how many times a file is 'cleaned' by adding or removing this line to the if statement in clean_that_data.py:

return_df_as_csv(build_classes(csv, lowest_similarity, ngram_size), saved_file_name)

csv -- data/raw_data/ .csv -- File you want to clean. It must be a .csv and must be in data/raw_data/

lowest_similarity -- float between 0 and 1 -- similarity threshold between two values in a column, values with similarity greater than lowest_similarity will become the same value.

ngram_size -- int (ideally between 2 and 4) -- size of character chunks used to assess similarity. i.e. ngram_size of 3 for similarity: ' si' 'sim' 'imi' 'mil' 'ila' 'lar' 'ari' 'rit' 'ity' 'ty '

About

Dashboard for displaying political donations.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • CSS 29.9%
  • SCSS 27.9%
  • HTML 21.1%
  • Python 13.2%
  • JavaScript 7.9%