Python

Today on my AWS Lambda Python function, its suffering from an error ValueError: Expecting property name: line 1 column 2 (char 1). This function receive a json events from AWS Kinesis and send it back to kafka. Apperently this is an error from json.loads if you have a json strings with a single quote >>> json_string = "{'name': rezha, 'ganteng': true}" >>> json.loads(json_string) # ValueError: Expecting property name: line 1 column 2 (char 1) To fix this, you could use ast....

Prioritize your queue A PriorityQueue is a type of queue imported from the module with the same name. It uses sort order to decide what to retrieve from it first (your object must have a way of comparing its instances): import queue class Rezha(object): def __init__(self, priority): self.priority = priority def __lt__(self, other): return self.priority > other.priority q = queue.PriorityQueue() q.put(Rezha(11)) q.put(Rezha(55)) q.put(Rezha(100)) while not q.empty(): print(q.get().priority) # output is 100 / 55 / 11 Having defined the __lt__ method, our PriorityQueue knows now how to sort elements of type Rezha....

Double ended queues with deque The deque class in the collections module makes it easy to create deques or double ended queues. Deques allow you to append and delete elements from both ends more efficiently than in lists. Import the module: from collections import deque Instantiate deque: d = deque() Append to right and left: d.append("b") d.appendleft("a") print(d) # output is: deque(['a', 'b']) In the same fashion, elements can be deleted (popped) :...

Best way to implement a simple queue A simple list can be easily used and implemented as a queue abstract data structure. A queue implies the first-in, first-out principle. However, this approach will prove inefficient because inserts and pops from the beginning of a list are slow (all elements need shifting by one). It’s recommended to implement queues using the collections.deque module as it was designed with fast appends and pops from both ends....

When you’re trying spark with its python repl, it’s really easy to write stuff using simple function or lambda. However, it will be a pain in the ass when you’re starting to try some complex stuff because you could easily miss something like indentation, etc. Try running your pyspark with this command IPYTHON_OPTS="notebook" path/to/your/pyspark It will start an IPython Notebook in your browser with Spark Context as sc variable. You could start using it like this: