Use Python's Django framework to write applications from Google AdSense to get reports _python

I have completed updating our real time revenue statistics in neutron. After I spent a week completing and updating our PHP script, I finally decided to start using Python to crawl, which was worth the time and energy I spent. I created a Django program that can store revenue statistics from different sources, and I can use these to go to the resume view and the API for statistical tools.

So in the past few days, I've written a script that can log in to other pages and crawl data, or, if they have an API, you can access the API directly. I found something.

1.requests >HTTPLIB2 (requests more than HTTPLIB2);

2.SOAP was bad, but it was at least a api,suds to make soap a little better. I learned that soap is the only API I've ever known that is completely based on. NET development.

3.Beautiful soup is a very good object of help;

4. I was really surprised that so many companies could survive such a crappy technology.

I saved Google Adsense, and they will have the best APIs and become the simplest implementations. He had more challenges than I had expected. Obviously you can't just insert username/password or apikey to get access to the API, you have to complete the Oauth2 handshake process.

Unfortunately, I found that the documentation was not as easy to query as I had hoped. I found a lot of dead links. I think Google's people should do better in this respect. For example, in their up to date developer Docs documentation, I found that they pointed out broken link to read more about authentication and authorization. (OK, how strange, I submitted the question as soon as possible, this link finally began to work, I guess you will thank me.) ）

So, this blog will try to record the process of getting reports from AdSense to my Django application.

To use the Google API to access AdSense reports, you need to use the AdSense Management API. This API only provides OAuth, so you need to complete the authentication process at least once in the browser to get your certificate, and then you can save the certificates for the next step. To tell you the truth, I have heard of oauth many times, but until now, I still do not need to use it in practice. So I am doing while learning, and welcome you to leave a message pointing out what I said is wrong.

As I know, Google has a huge API for all its products. Before you study adsense, you need to register your application with the Google API console. I have successfully registered my application. Since I do not yet have an available URL address, I now temporarily use my development URL (localhost:8000). It works as if it were normal. and use this link provided to download the JSON file.

Also, when you manage your APIs, you need to open the Service tab and open the AdSense Management API option. Otherwise, when you try to send the request, you get an error message "Access Not Configured".

Google has created a Python client library, and you can easily install the library via the PIP. It also contains a Django sample project that uses this library to implement the OAuth2 handshake process. I think it was written using Django 1.1 (since Django 1.5 was just released when writing this project), so it might be a bit outdated, but it's a good starting point.

My application is very simple. I just need to read the amount of revenue for the specified date and save it to my local database.

I created a new application in the Djaongo project, called "AdSense." and created a models.py file to store the authentication certificate.

From django.contrib.auth.models import User to
django.db import models from
oauth2client.django_orm Import Credentialsfield
 
class credential (models. Model):
  id = models. ForeignKey (User, primary_key=true)
  credential = Credentialsfield ()
 
class Revenue (models. Model):
  date = models. Datefield (unique=true)
  revenue = models. Decimalfield (max_digits=7, decimal_places=2)
 
  def __unicode__ (self): return
    ' {0} ${1} '. Format (self.date, Self.revenue)

I put the JSON file downloaded from the API console under my application's folder and created a views.py file

Import OS from django.conf import settings from django.contrib.auth.decorators import login_required from Django.contri B.sites.models Import Site from django.http import httpresponsebadrequest, HttpResponse from django.http import Httprespo Nseredirect from oauth2client import xsrfutil to oauth2client.client import flow_from_clientsecrets from Oauth2client.django_orm import Storage from. Models Import Credential Client_secrets = Os.path.join (Os.path.dirname (__ file__), ' client_secrets.json ') flow = Flow_from_clientsecrets (client_secrets, scope= ' https://www.googleapis.com/au Th/adsense.readonly ', redirect_uri= ' http://{0}/adsense/oauth2callback/'. Format (Site.objects.get_current (). Domain) @login_required def index (request): storage = Storage (credential, ' id ', Request.user, ' credential ') credent ial = Storage.get () If credential is-None or credential.invalid is true:flow.params[' state ' = xsrfutil.generate_to Ken (settings.
                        Secret_key,    Request.user) Authorize_url = Flow.step1_get_authorize_url () return Httpresponseredirect (Authorize_url) Else
 
: Return HttpResponse (' already validated. ') @login_required def auth_return (request): If not Xsrfutil.validate_token (settings. Secret_key, request. request[' state ', Request.user): Return httpresponsebadrequest () credential = Flow.step2_exchange (Request. REQUEST) storage = Storage (credential, ' id ', Request.user, ' credential ') storage.put (credential) return HttpResponse
 Redirect ("/")

In the urls.py file I included a URL file that links to my application

Main urls.py: From
 
django.conf.urls import patterns, include, url from
django.contrib import admin
 
Admin.autodiscover ()
 
urlpatterns = Patterns (
  ',
  url (r ' ^adsense/', include (' Adsense.urls ', namespace= ') AdSense "),
 
  url (r ' ^admin/doc/', include (' Django.contrib.admindocs.urls ')),
  URL (r ' ^admin/', include ( admin.site.urls))

adsense/urls.py: from
 
django.conf.urls import patterns, url
 
urlpatterns = Patterns (
  ' adsense.views ',
  url (r ' ^$ ', ' Index ', name= ' index '),
  url (r ' ^oauth2callback/$ ', ' auth_ Return ', name= ' Auth_return '),

Finally, you create a class that invokes the API and proceeds from a given date. It's on the adsense/tasks.py, because I'm going to take it as a task, hooked on CELERY/RABBITMQ.

Import datetime import Httplib2 from apiclient.discovery Import to django.contrib.auth.models import User from  Oauth2client.django_orm import Storage from. Models import credential, Revenue today = Datetime.date.today () yesterday = Today-datetime.timedelta (Days=1) class Scraper (object): Def get_report (self, start_date=yesterday, end_date=today ): User = User.objects.get (pk=1) storage = Storage (credential, ' ID ', user, ' credential ') credential = storage. Get () If not credential are None or credential.invalid is false:http = httplib2. HTTP () http = credential.authorize (http) service = build (' AdSense ', ' v1.2 ', http=http) reports = Service . Reports (), Reports.generate (Startdate=start_date.strftime ('%y-%m-%d '), Enddate=end_date.st Rftime ('%y-%m-%d '), dimension= ' DATE ', metric= ' earnings ',) data = Report.execute () for R ow in data[' rows ']: date = row[0] RevenuE = row[1] record = Revenue () try:r = Revenue.objects.get (date=date) pk = r.id except REVENUE.DOESNOTEXIST:PK = None Record.id = PK Record.date = Date Record.reve
 Nue = Revenue Record.save ()

To make it work, I open the http://localhost:8000/adsense/in the browser. This will require me to login to Google account. I am authorized to access AdSense for my application. The certificate is then saved in my local database, and then I can invoke the scraper Get_report () method. Congratulate me!. It can work well.