thumbnail
获取API数据,分享给更多的遵纪守法的我们

一、python爬取请求

1.分析API

先看下响应的数据,风险地区

在这里插入图片描述
请求的参数,也就要计算最后两个(签名与发送时间戳)

在这里插入图片描述
还有就是请求头里这两个是最重要的

在这里插入图片描述

开始翻代码,蓝色的是时间的计算,然后红色的是两个签名的计算方式

在这里插入图片描述

2.py爬虫

签名在手,数据你有,那么我们一起开始爬吧

o 为Python方式的时间计算 JS方式:((new Date).getTime() / 1e3).toFixed()

s1s2 分别为Python方式实现的两个签名的计算

签名1JS方式:CryptoJS.SHA256(e + i + a + e).toString(CryptoJS.enc.Hex).toUpperCase()

签名2JS方式:CryptoJS.SHA256(n + “fTN2pfuisxTavbTuYVSsNJHetwq5bJvCQkjjtiLM2dCratiA” + n).toString(CryptoJS.enc.Hex).toUpperCase()

import pymysql as MySQLdb
import requests
import hashlib
import time
import json

o = '%.3f' % (time.time() / 1e3)
e = o.replace('.','')
i = "23y0ufFl5YxIyGrI8hWRUZmKkvtSjLQA"
a = "123456789abcdefg"
s = "zdww"

def getSignature1():
    s = hashlib.sha256()
    s.update(str(e + i + a + e).encode("utf8"))
    b = s.hexdigest().upper()
    return b

def getSignature2():
    s = hashlib.sha256()
    s.update(str(e + 'fTN2pfuisxTavbTuYVSsNJHetwq5bJvCQkjjtiLM2dCratiA' + e).encode("utf8"))
    b = s.hexdigest().upper()
    return b

s1 = getSignature1()
s2 = getSignature2()

都算好了就开始拼数据发送了,下面是部分请求头与完整的请求参数

post_dict = {
    'appId': 'NcApplication',
    'key': '3C502C97ABDA40D0A60FBEE50FAAD1DA',
    'nonceHeader': '123456789abcdefg',
    'paasHeader': 'zdww',
    'signatureHeader': s1,
    'timestampHeader': e
}

headers = {
    'Referer': 'http://bmfw.www.gov.cn/',
    'Origin': 'http://bmfw.www.gov.cn',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0',
    'x-wif-nonce': 'QkjjtiLM2dCratiA',
    'x-wif-paasid': 'smt-application',
    'x-wif-signature': s2,
    'x-wif-timestamp': e,
    'Content-Type': 'application/json; charset=utf-8',
}

发送请求

url = 'http://103.66.32.242:8005/zwfwMovePortal/interface/interfaceJson'

def get_data():
    req = requests.post(url=url, data=json.dumps(post_dict), headers=headers)
    resp = req.text
    res = json.loads(resp)
    print(res['msg'])

    utime = res['data']['end_update_time']
    hcount = res['data']['hcount']
    mcount = res['data']['mcount']
    print(utime)
    print('高风险:' + str(hcount) + '个')
    print('中风险:' + str(mcount) + '个')

    hlist = res['data']['highlist']
    mlist = res['data']['middlelist']

    create_time = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())

	# 连你的库吧
    mysql_conn = MySQLdb.connect(host="", user="", passwd="", db="", port=3306, charset="UTF8")
    mysql_cursor = mysql_conn.cursor()

	# 可以查下当前更新时间段的数据是否已存在于数据库
    mysql_cursor.execute("SELECT count(*) - 1 AS count FROM sys_fx_address WHERE time = %(utime)s;",{'utime': utime})
    cres = mysql_cursor.fetchall()[0][0]
    print("此刷新时间已有数据:" + str(cres) + '条')

	# 更新统计数据
    mysql_cursor.execute(
        "update sys_fx_address set province = %(hcount)s, city = %(mcount)s, time = %(time)s, create_time = %(create_time)s where id = 1;",
        {'hcount': hcount, 'mcount': mcount, 'time': utime, 'create_time': create_time})

	# 开始遍历中高风险地区名单
    if cres <= 0:
        print("开始更新风险地区名单")
        for hd in hlist:
            type = hd['type']
            province = hd['province']
            city = hd['city']
            county = hd['county']
            area_name = hd['area_name']
            communitys = json.dumps(hd['communitys'], ensure_ascii=False)
            mysql_cursor.execute(
                "insert into sys_fx_address(type, province, city, county, area_name, time, communitys, create_time, level) "
                "values(%(type)s,%(province)s,%(city)s,%(county)s,%(area_name)s,%(time)s,%(communitys)s,%(create_time)s,%(level)s);",
                {'type': type, 'province': province, 'city': city, 'county': county, 'area_name': area_name,
                 'time': utime, 'communitys': communitys, 'create_time': create_time, 'level': 2})
            print(area_name)

        for md in mlist:
            type = md['type']
            province = md['province']
            city = md['city']
            county = md['county']
            area_name = md['area_name']
            communitys = json.dumps(md['communitys'], ensure_ascii=False)
            mysql_cursor.execute(
                "insert into sys_fx_address(type, province, city, county, area_name, time, communitys, create_time, level) "
                "values(%(type)s,%(province)s,%(city)s,%(county)s,%(area_name)s,%(time)s,%(communitys)s,%(create_time)s,%(level)s);",
                {'type': type, 'province': province, 'city': city, 'county': county, 'area_name': area_name,
                 'time': utime, 'communitys': communitys, 'create_time': create_time, 'level': 1})
            print(area_name)

    mysql_conn.commit()
    mysql_conn.close()

运行一下,将数据存入数据库

if __name__ == "__main__":
    get_data()
    print('---END---')

在这里插入图片描述

这里把乡镇一级的数据都存成数组形式的字符串了,前端可以直接取出来进行JSON转换

二、Fetch获取json文件

1.跨域解决

使用新浪提供的API: https://interface.sina.cn/news/wap/fymap2020_data.d.json

在 Vue 中访问会存在跨域问题,这里只是解决下这个问题

npm install fetch-jsonp 安装一下 fetchJsonp

import fetchJsonp from 'fetch-jsonp'

fetchJsonp('https://interface.sina.cn/news/wap/fymap2020_data.d.json', {
  'mode': 'cors',
  method: 'get',
  headers: {
    'content-type': 'application/json',
    credentials: 'include'
  }
}).then(res => {
  res.json().then(obj => {
    let d = obj.data
  })
}).then(data => {
  console.log('data', data)
}).catch(err => {
  console.log(err)
})

看了下数据,换别的API吧嘿嘿


下一篇