Enable Spark Context on Your Ipython Notebook
Sat Jun 25, 2016 · 76 words
Spark Python Ipython Notebook

When you’re trying spark with its python repl, it’s really easy to write stuff using simple function or lambda. However, it will be a pain in the ass when you’re starting to try some complex stuff because you could easily miss something like indentation, etc.

Try running your pyspark with this command

IPYTHON_OPTS="notebook" path/to/your/pyspark

It will start an IPython Notebook in your browser with Spark Context as sc variable. You could start using it like this:

back · he thought · he could · so he did · main