Dev (Sourcery refactored) #18

sourcery-ai · 2023-10-26T13:02:47Z

Pull Request #17 refactored by Sourcery.

If you're happy with these changes, merge this Pull Request using the Squash and merge strategy.

NOTE: As code is pushed to the original Pull Request, Sourcery will
re-run and update (force-push) this Pull Request with new refactorings as
necessary. If Sourcery finds no refactorings at any point, this Pull Request
will be closed automatically.

See our documentation here.

Run Sourcery locally

Reduce the feedback loop during development by using the Sourcery editor plugin:

Review changes via command line

To manually merge these changes, make sure you're on the dev branch, then run:

git fetch origin sourcery/dev
git merge --ff-only FETCH_HEAD
git reset HEAD^

Help us improve this pull request!

sourcery-ai

Due to GitHub API limits, only the first 60 comments can be shown.

sourcery-ai · 2023-10-26T13:02:49Z

Schedule/demo4.py


 def greet(name):
-    print('Hello {}'.format(name))
+    print(f'Hello {name}')


Function greet refactored with the following changes:

Replace call to format with f-string (use-fstring-for-formatting)

sourcery-ai · 2023-10-26T13:02:49Z

Schedule/demo8.py

 """
 def job1():
-    print("I'm running on threads %s" % threading.current_thread())
+    print(f"I'm running on threads {threading.current_thread()}")


Function job1 refactored with the following changes:

Replace interpolated string formatting with f-string (replace-interpolation-with-fstring)

sourcery-ai · 2023-10-26T13:02:50Z

Schedule/demo8.py

+    print(f"I'm running on threads {threading.current_thread()}")
 def job2():
-    print("I'm running on threads %s" % threading.current_thread())
+    print(f"I'm running on threads {threading.current_thread()}")


Function job2 refactored with the following changes:

Replace interpolated string formatting with f-string (replace-interpolation-with-fstring)

sourcery-ai · 2023-10-26T13:02:50Z

Schedule/demo8.py

+    print(f"I'm running on threads {threading.current_thread()}")
 def job3():
-    print("I'm running on threads %s" % threading.current_thread())
+    print(f"I'm running on threads {threading.current_thread()}")


Function job3 refactored with the following changes:

Replace interpolated string formatting with f-string (replace-interpolation-with-fstring)

sourcery-ai · 2023-10-26T13:02:50Z

chrome/chrome_helper.py

-CHROME_DRIVER_MAPPING_FILE = r"{}\mapping.json".format(CHROME_DRIVER_FOLDER)
-CHROME_DRIVER_EXE = r"{}\chromedriver.exe".format(CHROME_DRIVER_FOLDER)
-CHROME_DRIVER_ZIP = r"{}\chromedriver_win32.zip".format(CHROME_DRIVER_FOLDER)
+CHROME_DRIVER_MAPPING_FILE = f"{CHROME_DRIVER_FOLDER}\mapping.json"
+CHROME_DRIVER_EXE = f"{CHROME_DRIVER_FOLDER}\chromedriver.exe"
+CHROME_DRIVER_ZIP = f"{CHROME_DRIVER_FOLDER}\chromedriver_win32.zip"


Lines 13-15 refactored with the following changes:

Replace call to format with f-string [×3] (use-fstring-for-formatting)

sourcery-ai · 2023-10-26T13:02:51Z

crawler/proxy/proxy.py

-    ret = []
-    for i in range(1, page):
-        ret.append('http://www.samair.ru/proxy/proxy-%(num)02d.htm' % {'num': i})
-    return ret
+    return [
+        'http://www.samair.ru/proxy/proxy-%(num)02d.htm' % {'num': i}
+        for i in range(1, page)
+    ]


Function build_list_urls_3 refactored with the following changes:

Convert for loop into list comprehension (list-comprehension)

Inline variable that is immediately returned (inline-immediately-returned-variable)

sourcery-ai · 2023-10-26T13:02:51Z

crawler/proxy/proxy.py

-        ip = match[0] + "." + match[1] + match[2]
+        ip = f"{match[0]}.{match[1]}{match[2]}"


Function parse_page_3 refactored with the following changes:

Use f-string instead of string concatenation [×3] (use-fstring-for-concatenation)

sourcery-ai · 2023-10-26T13:02:51Z

crawler/proxy/proxy.py

-    ret = []
-    for i in range(1, page):
-        ret.append('http://www.pass-e.com/proxy/index.php?page=%(n)01d' % {'n': i})
-    return ret
+    return [
+        'http://www.pass-e.com/proxy/index.php?page=%(n)01d' % {'n': i}
+        for i in range(1, page)
+    ]


Function build_list_urls_4 refactored with the following changes:

Convert for loop into list comprehension (list-comprehension)

Inline variable that is immediately returned (inline-immediately-returned-variable)

sourcery-ai · 2023-10-26T13:02:51Z

crawler/proxy/proxy.py

-    ret = []
-    for i in range(1, page):
-        ret.append('http://www.ipfree.cn/index2.asp?page=%(num)01d' % {'num': i})
-    return ret
+    return [
+        'http://www.ipfree.cn/index2.asp?page=%(num)01d' % {'num': i}
+        for i in range(1, page)
+    ]


Function build_list_urls_5 refactored with the following changes:

Convert for loop into list comprehension (list-comprehension)

Inline variable that is immediately returned (inline-immediately-returned-variable)

sourcery-ai · 2023-10-26T13:02:51Z

crawler/proxy/proxy.py

-    ret = []
-    for i in range(1, page):
-        ret.append('http://www.cnproxy.com/proxy%(num)01d.html' % {'num': i})
-    return ret
+    return [
+        'http://www.cnproxy.com/proxy%(num)01d.html' % {'num': i}
+        for i in range(1, page)
+    ]


Function build_list_urls_6 refactored with the following changes:

Convert for loop into list comprehension (list-comprehension)

Inline variable that is immediately returned (inline-immediately-returned-variable)

sourcery-ai · 2023-10-26T13:02:54Z

crawler/proxy/proxy.py

+    type = -1  # 该网站未提供代理服务器类型
    for match in matches:
        ip = match[0]
        port = match[1]
-        type = -1  # 该网站未提供代理服务器类型


Function parse_page_6 refactored with the following changes:

Hoist statements out of for/while loops (hoist-statement-from-loop)

sourcery-ai · 2023-10-26T13:02:54Z

crawler/proxy/proxy.py

-    ret = []
-    for i in range(0, page):
-        ret.append('http://proxylist.sakura.ne.jp/index.htm?pages=%(n)01d' % {'n': i})
-    return ret
+    return [
+        'http://proxylist.sakura.ne.jp/index.htm?pages=%(n)01d' % {'n': i}
+        for i in range(0, page)
+    ]


Function build_list_urls_9 refactored with the following changes:

Convert for loop into list comprehension (list-comprehension)

Inline variable that is immediately returned (inline-immediately-returned-variable)

sourcery-ai · 2023-10-26T13:02:54Z

crawler/proxy/proxy.py

-        if (type == 'Anonymous'):
-            type = 1
-        else:
-            type = -1
+        type = 1 if (type == 'Anonymous') else -1


Function parse_page_9 refactored with the following changes:

Replace if statement with if expression (assign-if-exp)

sourcery-ai · 2023-10-26T13:02:54Z

crawler/proxy/proxy.py

-    ret = []
-    for i in range(1, page):
-        ret.append('http://www.publicproxyservers.com/page%(n)01d.html' % {'n': i})
-    return ret
+    return [
+        'http://www.publicproxyservers.com/page%(n)01d.html' % {'n': i}
+        for i in range(1, page)
+    ]


Function build_list_urls_10 refactored with the following changes:

Convert for loop into list comprehension (list-comprehension)

Inline variable that is immediately returned (inline-immediately-returned-variable)

sourcery-ai · 2023-10-26T13:02:54Z

crawler/proxy/proxy.py

-    ret = []
-    for i in range(1, page):
-        ret.append('http://www.my-proxy.com/list/proxy.php?list=%(n)01d' % {'n': i})
-
-    ret.append('http://www.my-proxy.com/list/proxy.php?list=s1')
-    ret.append('http://www.my-proxy.com/list/proxy.php?list=s2')
-    ret.append('http://www.my-proxy.com/list/proxy.php?list=s3')
+    ret = [
+        'http://www.my-proxy.com/list/proxy.php?list=%(n)01d' % {'n': i}
+        for i in range(1, page)
+    ]
+    ret.extend(
+        (
+            'http://www.my-proxy.com/list/proxy.php?list=s1',
+            'http://www.my-proxy.com/list/proxy.php?list=s2',
+            'http://www.my-proxy.com/list/proxy.php?list=s3',
+        )
+    )


Function build_list_urls_11 refactored with the following changes:

Merge consecutive list appends into a single extend [×2] (merge-list-appends-into-extend)

Convert for loop into list comprehension (list-comprehension)

sourcery-ai · 2023-10-26T13:02:55Z

crawler/src/mul_spider.py

-    urls = []
-    for page in range(beg, end):
-        urls.append('http://www.baidu.com?&page=%d' % page)
+    urls = ['http://www.baidu.com?&page=%d' % page for page in range(beg, end)]


Function run_spider refactored with the following changes:

Convert for loop into list comprehension (list-comprehension)

sourcery-ai · 2023-10-26T13:02:55Z

crawler/src/mul_spider.py

-    res = []
-    for i in range(len(s) - 1):
-        res.append((s[i], s[i + 1]))
+    res = [(s[i], s[i + 1]) for i in range(len(s) - 1)]


Function main refactored with the following changes:

Convert for loop into list comprehension (list-comprehension)

sourcery-ai · 2023-10-26T13:02:55Z

crawler/src/proxy_req.py

-    proxies = {
-        'http': '{proxy_type}://{ip}:{port}'.format(proxy_type=proxy_type, ip=ip, port=port),
-        'https': '{proxy_type}://{ip}:{port}'.format(proxy_type=proxy_type, ip=ip, port=port),
+    return {
+        'http': '{proxy_type}://{ip}:{port}'.format(
+            proxy_type=proxy_type, ip=ip, port=port
+        ),
+        'https': '{proxy_type}://{ip}:{port}'.format(
+            proxy_type=proxy_type, ip=ip, port=port
+        ),
    }
-    return proxies


Function get_proxy_dict refactored with the following changes:

Inline variable that is immediately returned (inline-immediately-returned-variable)

sourcery-ai · 2023-10-26T13:02:55Z

crawler/src/sync_spider.py

-    urls = []
-    for page in range(1, 1000):
-        urls.append('http://www.jb51.net/article/%s.htm' % page)
+    urls = [f'http://www.jb51.net/article/{page}.htm' for page in range(1, 1000)]


Function main refactored with the following changes:

Convert for loop into list comprehension (list-comprehension)

Replace interpolated string formatting with f-string (replace-interpolation-with-fstring)

sourcery-ai · 2023-10-26T13:02:55Z

crawler/src/test.py

 class TestSyncSpider(SyncSpider):
    def handle_html(self, url, html):
        print(html)
-        pass


Function TestSyncSpider.handle_html refactored with the following changes:

Remove redundant pass statement (remove-redundant-pass)

sourcery-ai · 2023-10-26T13:02:58Z

crawler/src/test.py

 class TestAsyncSpider(AsyncSpider):
    def handle_html(self, url, html):
        print(html)
-        pass


Function TestAsyncSpider.handle_html refactored with the following changes:

Remove redundant pass statement (remove-redundant-pass)

sourcery-ai · 2023-10-26T13:02:58Z

crawler/src/test.py

-urls = []
-for page in range(1, 1000):
-    #urls.append('http://www.jb51.net/article/%s.htm' % page)
-    urls.append('http://www.imooc.com/data/check_f.php?page=%d'%page)
+urls = [
+    'http://www.imooc.com/data/check_f.php?page=%d' % page
+    for page in range(1, 1000)
+]


Lines 24-27 refactored with the following changes:

Convert for loop into list comprehension (list-comprehension)

This removes the following comments ( why? ):

#urls.append('http://www.jb51.net/article/%s.htm' % page)

sourcery-ai · 2023-10-26T13:02:58Z

crawler/src/xpath_utils.py

-        func_name = 'get_' + field
+        func_name = f'get_{field}'
        xpath_str = self.xpath_dict.get(field)
        if hasattr(self, func_name):
            return getattr(self, func_name)(xpath_str)
-        else:
-            self.logger.debug(field, self.url)
-            return self.parser.xpath(xpath_str)[0].strip() if xpath_str else ''
+        self.logger.debug(field, self.url)
+        return self.parser.xpath(xpath_str)[0].strip() if xpath_str else ''


Function XpathCrawler.get_field refactored with the following changes:

Use f-string instead of string concatenation (use-fstring-for-concatenation)

Remove unnecessary else after guard condition (remove-unnecessary-else)

sourcery-ai · 2023-10-26T13:02:58Z

crawler/src/xpath_utils.py

-        xpath_result = {}
-        for field, xpath_string in self.xpath_dict.items():
-            xpath_result[field] = makes(self.get_field(field))    # to utf8
-        return xpath_result
+        return {
+            field: makes(self.get_field(field))
+            for field, xpath_string in self.xpath_dict.items()
+        }


Function XpathCrawler.get_result refactored with the following changes:

Convert for loop into dictionary comprehension (dict-comprehension)

Inline variable that is immediately returned (inline-immediately-returned-variable)

This removes the following comments ( why? ):

# to utf8

sourcery-ai · 2023-10-26T13:02:58Z

crawler/src/xpath_utils.py

-        category_urls = []
-        for href in category_hrefs:
-            category_urls.append(urljoin(self.domain, href))
+        category_urls = [urljoin(self.domain, href) for href in category_hrefs]


Function FantasyhairbuySite.generate_category_urls refactored with the following changes:

Convert for loop into list comprehension (list-comprehension)

sourcery-ai · 2023-10-26T13:02:59Z

file_operation/delete_files.py

-    backup_name = os.path.basename(file_path) + '_' + datetime.datetime.now().strftime('%Y%m%d%H%M%S')
+    backup_name = (
+        f'{os.path.basename(file_path)}_'
+        + datetime.datetime.now().strftime('%Y%m%d%H%M%S')
+    )


Function backup_file refactored with the following changes:

Use f-string instead of string concatenation (use-fstring-for-concatenation)

sourcery-ai · 2023-10-26T13:03:00Z

file_operation/delete_files.py

-            backup_list = glob.glob(os.path.join(path, pattern + '_*'))
+            backup_list = glob.glob(os.path.join(path, f'{pattern}_*'))


Function main refactored with the following changes:

Use f-string instead of string concatenation (use-fstring-for-concatenation)

sourcery-ai · 2023-10-26T13:03:00Z

json_loads/json_loads.py

-
-    # if it's anything else, return it in its original form
-    return data
+    return data.encode('utf-8') if str(type(data)) == "<type 'unicode'>" else data


Function _byteify refactored with the following changes:

Lift code into else after jump in control flow (reintroduce-else)

Replace if statement with if expression (assign-if-exp)

This removes the following comments ( why? ):

# if it's anything else, return it in its original form

sourcery-ai · 2023-10-26T13:03:00Z

leancloud/leancloud_api.py

        try:
-            res = query.find()
-            return res
+            return query.find()


Function LeanCloudApi.get_skip_obj_list refactored with the following changes:

Inline variable that is immediately returned (inline-immediately-returned-variable)

sourcery-ai · 2023-10-26T13:03:00Z

leancloud/leancloud_api.py

-        img_info_url = img_url + '?imageInfo'
+        img_info_url = f'{img_url}?imageInfo'


Function LeanCloudApi.add_img_info refactored with the following changes:

Use f-string instead of string concatenation (use-fstring-for-concatenation)

'Refactored by Sourcery'

bfcfdbf

sourcery-ai bot force-pushed the sourcery/dev branch from 290fbe5 to bfcfdbf Compare October 26, 2023 13:02

sourcery-ai bot requested a review from jumploop October 26, 2023 13:02

sourcery-ai bot commented Oct 26, 2023

View reviewed changes

		ip = match[0] + "." + match[1] + match[2]
		ip = f"{match[0]}.{match[1]}{match[2]}"

		backup_list = glob.glob(os.path.join(path, pattern + '_*'))
		backup_list = glob.glob(os.path.join(path, f'{pattern}_*'))

		img_info_url = img_url + '?imageInfo'
		img_info_url = f'{img_url}?imageInfo'

Dev (Sourcery refactored) #18

Are you sure you want to change the base?

Dev (Sourcery refactored) #18

Uh oh!

Conversation

sourcery-ai bot commented Oct 26, 2023

Uh oh!

sourcery-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

sourcery-ai bot Oct 26, 2023