Weird bug in the Integration API. If you merge a pull request by force-pushing, like this, the PR will be marked as merged, but will not be closed:
curl -X PATCH -H "Authorization: token v1.CENSORED" https://api.github.com/repos/notriddle/test_repo/git/refs/heads/master -d '{"sha":"7bd3e0daa0f8a26614fa7788b363f7c8c6d09d6f","force":true}'
[where 7bd3e0daa0f8a26614fa7788b363f7c8c6d09d6f
is the commit at the head of the pull request]
The pull request will be marked as merged, but will not be closed. This only happens when you're acting as an Integration: if you use a Personal Access Token, it will work correctly.
Screenshot for permanent proof:
And a link to prove I didn't just photoshop it: https://github.com/notriddle/test_repo/pulls