Google - Chuck File Download

This example is discussed in stack overflow, tkx #stackoverflow.

#answer: https://stackoverflow.com/a/39225039

PYTHON

import requests

def download_file_from_google_drive(id, destination):
    URL = "https://docs.google.com/uc?export=download"

    session = requests.Session()

    response = session.get(URL, params = { 'id' : id }, stream = True)
    token = get_confirm_token(response)

    if token:
        params = { 'id' : id, 'confirm' : token }
        response = session.get(URL, params = params, stream = True)

    save_response_content(response, destination)    

def get_confirm_token(response):
    for key, value in response.cookies.items():
        if key.startswith('download_warning'):
            return value

    return None

def save_response_content(response, destination):
    CHUNK_SIZE = 32768

    with open(destination, "wb") as f:
        for chunk in response.iter_content(CHUNK_SIZE):
            if chunk: # filter out keep-alive new chunks
                f.write(chunk)

 

 

RUN

file_id = '0B1fGSuBXAh1IeEpzajRISkNHckU'
destination = '/home/myusername/work/myfile.ext'
download_file_from_google_drive(file_id, destination)

 

 

 

Apache Kafka - Intall & Setup in Windows

STEPS

1. Download kafka_2.12-2.5.0.zip and extract it [For ex: D:/BigData/kafka_2.12-2.5.0 ]
2. Kafka itself comes with zookeeper, so not necessary to download Zookeeper seperately.
3. Go to Command prompt D:/BigData/kafka_2.12-2.5.0

4. Start zookeeper
5. Start kafka
6. Create Topic
7. Start producer "client" to send data
8. Start consumer "server" to receive those data


Commands from 4th to 8th.

4.  D:\..> start zookeeper-server-start.bat D:\BigData\kafka_2.12-2.5.0\config\zookeeper.properties

5.  D:\..>start kafka-server-start.bat D:\BigData\kafka_2.12-2.5.0\config\server.properties

6.  D:\..>start kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testtopic

7.  D:\..>start kafka-console-producer.bat --broker-list localhost:9092 -topic testtopic

8. D:\..>start kafka-console-consumer.bat --bootstrap-server localhost:9092 -topic testtopic

Now type something in 7th - CMD prompt.
See the received output in 8th CMD prompt.

All set go.

Note: First stop kafka and then zookeeper. else kafka will keep retrying..