Rezha Julio

The Hard Coded Chemist

Enable Spark Context on Your Ipython Notebook


When you’re trying spark with its python repl, it’s really easy to write stuff using simple function or lambda. However, it will be a pain in the ass when you’re starting to try some complex stuff because you could easily miss something like indentation, etc.

Try running your pyspark with this command

IPYTHON_OPTS="notebook" path/to/your/pyspark

It will start an IPython Notebook in your browser with Spark Context as sc variable. You could start using it like this: