Chapter 11 - Processing the Rows of a DataFrame

See also the corresponding course notes, here.

View a printable version of these slides here.

Reasons to avoid loops

• Readability
  • loops are typically more code
  • sometimes the meaning of the code is clearer with a loop, sometimes clearer other ways
• Speed
  • loops are typically slower than other options

Using apply() instead

• Usually requires writing the function you want to apply, but not always
  • Sometimes the function is built in (like math.sqrt)
  • Sometimes it's short enough for a lambda, as in lambda name: 'Mr.' in name
• Mostly the same as map()
  • Some differences detailed in the course notes
  • Most importantly, map() supports Python dictionaries
  • So does replace() which doesn't support functions...so many options!
• Prepares us for easy parallel processing with swifter or multiprocessing

Map-reduce

Exercise 1 - Historic precipitation

View the notebook exercise-1.ipynb in this Deepnote project.

Duplicate the project and answer Questions 1-4 in the Exercise 1 notebook.

Your instructor will explain how we will present and discuss our answers together.

Split-apply-combine

Exercise 2 - politics and real estate

View the notebook exercise-2.ipynb in your own copy of the same Deepnote project.

Answer Questions 1-4 in the Exercise 2 notebook.

Your instructor will explain how we will present and discuss our answers together.

Exercise 3 - manufacturing

View the notebook exercise-3.ipynb in your own copy of the same Deepnote project.

It asks you to envision an imaginary scenario of several data tables. You have to answer Questions 1-3 with plans for how you would write code to answer questions in that situation, using either the map-reduce or split-apply-combine paradigms, as needed.

No actual coding is required.

Your instructor will explain how we will present and discuss our answers together.