This was the second time I have been there

(Grammar Questions I Couldn't Answer)

A passage from a student writing:

We went to Dalat last year.  It was the second time I have been there....

To me, the present perfect in "I have been there" is sounding wrong.  I want to put it into either the past simple, or the past perfect.
But why is the present perfect wrong here?  What rule is it violating?
Ordinarily, we teach students that we use the past simple with a specific time reference, and the present perfect without time references.
e.g.
I have been to Japan.
I went to Japan last year.

Here, does "the second time" count  as a time reference?  Is that why we can't use the present perfect?

Or, is my native speaker intuition off here?  Is the present perfect actually okay to use here?

Update
I ran this by my manager for a second opinion.

My manager thought that the present perfect was definitely wrong, and that it should definitely be in the past perfect.

He tried to break it down for me.
"We went to Dalat--past tense" he said.  "It was the second time I had been there--referring to the time before the past tense."

But I was resistant to this explanation.  "No, It was the second time is synonymous with we went to Dalat," I argued.  "They're both referring to the same time.  Of course, the fact that it was the second time implies that there was also a first time.  But that first time isn't in the sentence."

"No, I'll draw you a time line," my manager said.

My manager agreed with me that  "We went to Dalat" and "it was the second time" were referring to the same event, so he located both clauses on the same point on the time line.
However my manager argued "(that) I had been there" was referring to the first time, and so he located it further back on the time line.

I was still resisting this.  I thought "(that) I had been there" was modifying "the second time" and thus still referring to the second time.

And yet, I did have to agree that, in terms of my native speaker intuition,  my manager's version sounded correct.

"We went to Dalat last year.  It was the second time I had been there" sounded better than "We went to Dalat last year.  It was the second time I have been there."

My manager also pointed out that in the present tense, the sentence would read: "I am in Dalat.  It is the second time I have been here."  His logic is that in the present tense, "(that) I have been here" is used to refer to a point in the unspecified past that has an effect on the present.
Once again, I'm skeptical of the reasoning.  I think "(that) I have been here" is modifying "it is the second time".
And yet, my manager is obviously right that the sentence sounds correct in the present perfect.