Cannot run script

[Tiemichael]

Hi,
I am unable to run the script on my Raspberry 4 running ArchLinux.
When I start the script via:
(.venv) [michael@nanoraspi exportMediaWiki2HTML]$ python3 exportMediaWiki2Html.py --url=https://wiki.xxxxx.com --user=min@mw2html --password=08g24xxxxxxmv7072f66qf4df7hpabq
(user, password are Bot-user and pwd)

I receive:
`Traceback (most recent call last):
File "/home/michael/exportMediaWiki2HTML/.venv/lib/python3.10/site-packages/requests/models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python3.10/json/init.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/michael/exportMediaWiki2HTML/exportMediaWiki2Html.py", line 114, in
DATA = R.json()
File "/home/michael/exportMediaWiki2HTML/.venv/lib/python3.10/site-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
`

Any idea what is wrong?
Thanks!

[tpokorra]

Hello @Tiemichael ,

please add a new line before line 114 in exportMediaWiki2Html.py, and insert a print command.
It should look like this:

  R = S.post(url + "/api.php", data=PARAMS_1)
  print(R.content)
  DATA = R.json()

That might give you an idea.
Looking at the code, I wonder about the warning, that action=login is deprecated. Which version of MediaWiki are you using?

Perhaps I need to look at https://www.mediawiki.org/wiki/API:Login#Method_2._clientlogin

[Tiemichael]

Thanks for you quick reply!

I saw you updated the py-script, so I pulled it, and now error abut "requests/models.py" is gone.
The script prints the message: "cannot parse json from action:login" followed by html-file "MediaWiki API help - MT-Wiki"

If I add the print-command, this html-file is displayed twice.

MediaWiki version is: 1.39.2

PS: I might have missed an important hint at the end of the html:

<script>(RLQ=window.RLQ||[]).push(function(){mw.log.warn("This page is using the deprecated ResourceLoader module \\"mediawiki.skinning.interface\\".\\n[1.37] The use of the `content` feature with ResourceLoaderSkinModule is deprecated. Use `content-media` instead. [1.37] The use of the `legacy` feature with ResourceLoaderSkinModule is deprecated(T89981) and is a NOOP since 1.39 (T304325). This should be urgently omited to retain compatibility with future MediaWiki versionsMore information can be found at [[mw:Manual:ResourceLoaderSkinModule]]. ");mw.config.set({"wgBackendResponseTime":1259});});</script>

PPS: here is another finding:
If I use my local-url (http://192.168....) the program saves the Main-page - asking for Login. This ONLY happens if I provide a VALID Bot-user and password.
I checked my NGIX-settings, but they are identical to the ones found at the internet. (To access the page via https, I use a (NGINX) reverse proxy on an other server - which works fine for 'normal' operations.

PPS:: it seems that access to the API via browser is working fine for both external (https://wiki.xxx.com.) and internal (192.168.1...) access. Commands like https://wiki.xxxx.com/api.php?action=query&meta=authmanagerinfo&amirequestsfor=login show a proper response

[tpokorra]

I have now upgraded my wiki from 1.35 LTS to 1.39.3 LTS.
My hope was that I get the same error as you.
But it works for me.

I am not using a special bot user for calling the script, but my normal user:

python3 exportMediaWiki2Html.py -l https://mywiki.pokorra.de -u "myusername" -p "mypassword"

Ok, now I understand the warning Main-account login via "action=login" is deprecated and may stop working without warning. To continue login with "action=login", see [[Special:BotPasswords]]. So it is still working to login with my normal user, but it is not supported.

So I created a new Bot at https://mywiki.pokorra.de/index.php?title=Spezial:BotPasswords

It is now working for me without a warning:

python3 exportMediaWiki2Html.py -l https://mywiki.pokorra.de -u "Timo@ExportBot" -p "generated_secret"

(Timo is my username, and ExportBot is the name of the bot)

@Tiemichael
I wonder why the main page got downloaded with the login form.
Anything special in your -l parameter? or are there any redirects happening in your .htaccess file?

[Tiemichael]

Hi,
thank you for your help!
Too bad, but despite the changes my results have hardly changed.
(Note, I have added a couple of comments to the code, for easier debugging)

If I use:
python3 exportMediaWiki2Html.py -l https://wiki.xxxxxx.com/ -u Admin@mw2html -p my_secret_password,
I receive
URL: https://wiki.xxxxxx.com/
DATA: {'batchcomplete': '', 'query': {'tokens': {'logintoken': '0ddcd3c5cedbd4c6c61888bb1aa745b66476d3e0+\\'}}}
LOGIN TOKEN: 0ddcd3c5cedbd4c6c61888bb1aa745b66476d3e0+\
PARAMS_1: {'action': 'login', 'lgname': 'Admin@mw2html', 'lgpassword': 'my_secret_password', 'lgtoken': '0ddcd3c5cedbd4c6c61888bb1aa745b66476d3e0+\\', 'format': 'json'}
R: <Response [200]>
cannot parse json from action:login
b'<!DOCTYPE html>\n<html class="client-nojs" lang="en" dir="ltr">\n<head>\n<meta charset="UTF-8"/>\n<title>MediaWiki API help - MT-Wiki</title>\ ..............
The response is independent of the lgpassword I use.

If I use a local-ip with
python3 exportMediaWiki2Html.py --url=http://10.112.59.101:8085 -u "Admin@mw2html" -p my_secret_password
I receive:
URL: http://10.112.59.101:8085
DATA: {'batchcomplete': '', 'query': {'tokens': {'logintoken': '3553eabc9210270c4d66d9d7019363a26476d5f3+\\'}}} LOGIN TOKEN: 3553eabc9210270c4d66d9d7019363a26476d5f3+\
PARAMS_1: {'action': 'login', 'lgname': 'Admin@mw2html', 'lgpassword': 'my_secret_password', 'lgtoken': '3553eabc9210270c4d66d9d7019363a26476d5f3+\\', 'format': 'json'}
R: <Response [200]>
{'pageid': 1, 'ns': 0, 'title': 'Main Page'}

The saved page is the 'login-prompt.

If I use a wrong password here, I get:
R: <Response [200]>
{'error': {'code': 'readapidenied', 'info': 'You need read permission to use this module.'

I checked my server, I don't have any .htaccess-file

[tpokorra]

so it works locally, but not from the outside. that is strange.
Is there any proxy in front of your website?

Can you try to access your site with the bot account from outside with the tool curl, and see the debug messages?

[tpokorra]

another idea: which IP Address range did you set for your bot?

to allow from everywhere, use when creating the bot:

0.0.0.0/0
::/0

[Tiemichael]

so it works locally, but not from the outside. that is strange. Is there any proxy in front of your website?

I would not say it works locally, as it only opens a page asking for login as also shown in issue #30
My overall system-configuration:

Internet <==> Router <==> Server1 <==> Server2
Router forwards port 443 to Server1
Server1 is Reverse Proxy and 'forwards' port 8085 to HTTPS
Server2 runs the Wiki, with output to port 8085
(in general is configuration is running with no problem)

Can you try to access your site with the bot account from outside with the tool curl, and see the debug messages?

I am not really familiar with curl, but when running a CMD-prompt on my PC:
curl -v -u Admin@mw2html:my_secret_password https://wiki.xxxxx.com
I receive
* Trying 112.xxx.145.72:443...
* Connected to wiki.xxxxx.com (112.xxx.145.72) port 443 (#0)
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* ALPN: server accepted http/1.1
* using HTTP/1.1
* Server auth using Basic with user 'Admin@mw2html'
> GET / HTTP/1.1
> Host: wiki.xxxxx.com
> Authorization: Basic QWRtaW5AbXcyaHRtbDpibHRoOXJvNGowOGVycTIxbzA0YWZnZjBqMzliN3Vwdg==
> User-Agent: curl/8.0.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.18.0
< Date: Fri, 02 Jun 2023 08:13:11 GMT
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Powered-By: PHP/8.2.6
< X-Content-Type-Options: nosniff
< Content-language: en
< X-Frame-Options: DENY
< Vary: Accept-Encoding, Cookie, Authorization
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< X-Request-Id: cfcb7bf48724bd3c3e5ae6f9

followed by a HTML-page starting with:
<title>Login required - MT-Wiki</title>
This is independent of the password provided.

My ip-address range is:
0.0.0.0/0
::/0